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ABSTRACT 

Although science educators at all levels have focused on teaching students scientific literacy for nearly five decades, studies 
indicate that the average student remains far from scientifically literate. To address this issue at the local level, faculty at Trinity 
University, in San Antonio, Texas, significantly revised the curriculum of an existing introductory physical geology laboratory 
course. The course, which satisfies general education requirements at Trinity, was revised to provide students learning 
opportunities in a scientific process context as part of a new science literacy initiative. This effort was spurred by general 
dissatisfaction with the existing curricular structure of the course as well as a new interdisciplinary. National Science 
Foundation (NSF)-funded initiative to support the integration of research-grade instrumentation in curricula and 
undergraduate research across campus. The physical geology laboratory course revision was based on research that 
demonstrated the efficacy of learning through active participation, interpretation, iteration, and reflection, especially when 
knowledge and skills are gained within an explicit scientific process context. In addition to significantly revising laboratory 
activities, we added new activities within the course framework that involved the use of two new, NSF-funded instruments, 
including a handheld X-ray fluorescence spectrometer (XRF) and an inductively coupled plasma-optical emissions 
spectrometer (ICP-OES), which we used to improve student understanding of qualitative and quantitative elemental 
analyses. Finally, we introduced the use of a new course reader that provided both background materials for each activity as 
well as a new focus on providing a scientific process context for students. To assess student learning, we used in-class 
observations, student-instructor discussions, pre- and postlearning questionnaires, prelaboratory quizzes, course activities 
completed during class time, modified postactivity reflection questions, practical examinations, and a final examination. We 
also included faculty, staff, and administrator perspectives to qualitatively assess the impact of course changes upon student 
learning. Our results imply that students achieved the primary learning goals we developed for this scientific literacy initiative, 
including: (1) improved understanding of the scientific process and the nature of science; (2) improved understanding of 
qualitative and quantitative elemental research methods; and (3) improved understanding of the applicability of scientific 
research to real-world problems. Importantly, our findings suggest that the integration of research-grade instrumentation into 
introductory coursework, in a scientific process context, is an effective way to promote scientific literacy as well as to provide 
opportunities for students to understand and apply the knowledge and skills necessary to perform scientific research. We 
believe that this example of a significant course redesign provides a model that can be transferred to other geosciences 
departments as well as to other scientific disciplines. © 2014 National Association of Geoscience Teachers. [DOI: 10.5408/13- 
073.1] 
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INTRODUCTION 

Introductory laboratory courses play an important role 
in geoscience teaching at most colleges and universities. 
Traditionally, these courses provide learning opportunities 
with some hands-on activity but are primarily focused on 
memorization of content without providing students the 
opportunity to participate directly in the practice of active 
science research (e.g., Latour, 1987). This represents a 
significant missed opportunity: since many of these students 
will not study science beyond these introductory courses, 
these laboratory activities may represent their only direct 
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experiences with a scientist (the instructor). We believe that 
the introductory science laboratory is the ideal time to 
engage students with focused, hands-on, scientific investi¬ 
gation and discovery that will encourage some to pursue 
scientific research as a career and provide others with an 
authentic foundation for life-long scientific literacy. 

The advantages of hands-on or project-based learning 
are well supported in the literature (e.g., Blosser, 1983; Baird, 
1990; Bybee, 2000), and Nelson et al. (2010) found that 
students enrolled in university-level geoscience laboratory 
sections performed significantly better in an associated 
lecture course than their colleagues who were enrolled in 
the lecture course alone. Additionally, Richter-Egger et al. 
(2010) found improved student attitudes about science as 
well as improved learning related to the integration of 
instrument-based research into introductory science labora¬ 
tories. Finally, Nakhleh and Krajcik (1993, 1994) found a 
positive correlation between the use of instrumentation and 
student understanding of specific chemical phenomena. 

While the literature suggests a clear learning advantage 
is gained with hands-on laboratory activities, even when 
students collect qualitative and quantitative data and 
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perform basic interpretations of those data in laboratory 
courses, they do not necessarily recognize this as participa¬ 
tion in the scientific research process (e.g., Ryder et al., 1999; 
Schwartz et al., 2004). Introducing the use of new 
instrument-based activities into the laboratory curriculum 
certainly yields a positive impact on student learning and 
attitudes about science, but in order for students to achieve a 
much fuller understanding of the value of the scientific 
process and the nature of science research, it is also 
necessary to make their participation in the scientific 
processes explicit (e.g., Lederman, 2007) and permit them 
time to reflect on that participation (e.g., Schwartz et al., 
2004; Carpi and Egger, 2010). 

With these notions in mind, and supported by a 
National Science Foundation grant (award #0942940), the 
geology and chemistry departments at Trinity University 
collaboratively developed a new scientific literacy initiative. 
This project began with the integration of research-grade 
instrumentation into existing introductory geoscience labo¬ 
ratory activities, while at the same time providing students a 
significantly revised curricular framework to explicitly 
emphasize many components of the scientific process and 
the nature of science, more generally. In this paper, we 
report on the development and implementation of course 
materials and curriculum associated with this initiative, 
student engagement before and after implementation, and 
our assessment of ultimate learning outcomes. The findings 
of this study support a new curriculum model that should 
improve students' overall scientific literacy and positively 
impact overarching departmental and institutional goals. 

WHAT IS SCIENTIFIC LITERACY? 

Although the term "scientific literacy" is not well 
defined in the literature (Deng et al., 2011, and references 
therein), Durant (1993) claimed that at its simplest, scientific 
literacy is "what the general public ought to know about 
science," (p. 129) while Jenkins (1994) more closely defined 
scientific literacy as implying "an appreciation of the nature, 
aims, and general limitations of science, coupled with some 
understanding of the more important scientific ideas" (p. 
5345; see also Laugksch, 2000). Alternatively, the American 
Association for the Advancement of Science defined a 
scientifically literate person as someone who is familiar with 
the natural world, understands key concepts and principles 
of science, has the capacity for scientific reasoning, and is 
able to use scientific knowledge for personal and social 
purposes (AAAS Project 2061 et al., 1994; Jurecki and 
Wander, 2012). More relevant to introductory geoscience 
courses, the Earth Science community recently produced 
four major scientific literacy documents (Atmospheric 
Science Literacy Framework, 2007; U.S. Climate Change 
Science Program, 2009; Earth Science Literacy Initiative, 
2010; Wysession et al., 2012; Ocean Literacy Framework, 
2013) that provide summaries of "big ideas" that can be 
utilized by instructors at all levels to more effectively teach 
scientific literacy. 

No matter the definition or framework for scientific 
literacy, an understanding of the scientific process is 
considered a vital component of scientific literacy (e.g., 
Millar and Osborne, 1998; Abd-El-Khalick and Lederman, 
2000; Lederman, 2007; Deng et al., 2011), and Driver et al. 
(1996) stressed the personal and societal benefits of 


producing students who have sophisticated views of the 
nature of science. Importantly, Miller (2010a, 2010b, 2012) 
has clearly shown that exposure to college science courses is 
a strong predictor of civic scientific literacy in adults of all 
ages, so it is likely that all students who complete an 
introductory geoscience course will become more scientifi¬ 
cally literate, regardless of pedagogy. However, since a 
student's ideas about the nature of science are constructed 
largely from his or her experiences in science classrooms and 
laboratories, and/or within other appropriate teaching 
contexts, students' views of science can be significantly 
developed and broadened (e.g., Ryder et al., 1999; Van Eijck 
et al., 2008) in well-designed college science laboratory 
courses. 

The Scientific Process and the Nature of Science 

Broadly held student misconceptions about the nature of 
science (e.g., McComas, 1998) provide additional challenges 
to our effort. Students commonly enter an introductory 
geoscience classroom with many misconceptions (best 
summarized in McComas, 1998), which include the ideas 
that the scientific process consists of a set of tasks that is 
prescribed and linear and is the same for all scientists (i.e., 
the scientific method), cannot involve creativity or innova¬ 
tion, begins with an educated guess about a given 
phenomenon (hypothesis formation), and must include 
experimentation to test that single hypothesis. The average 
student also thinks that once the hypothesis holds true for a 
few more experiments, the scientist has now developed a 
theory. Finally, if this theory can be proven true, it becomes a 
law, which is absolute. These inaccurate ideas have been 
introduced by instructors and reinforced in textbooks and 
classrooms at all levels of science education (e.g., McComas, 
1998). 

Although there is debate about what the "nature of 
science" should be (e.g.. National Science Teachers Associ¬ 
ation, 1982; AAAS, 1993; NRC, 1996; Deng et al., 2011), 
there is some consensus about the nature of science among 
science educators (Deng et al., 2011), which is summarized 
by Lederman (2007). Lederman (2007) proposes that 
scientific knowledge: (1) is subject to change; (2) is based 
on or derived from observations of the natural world; (3) is 
subjective; (4) involves human imagination, inference, and 
creativity; and (5) is socially and culturally embedded. 
Further, Lederman (2007) includes the definitions of and 
relations between theories and laws as well as understand¬ 
ing the difference between observations and inferences. 

Leach (2002) emphasized five broad areas that affect 
students' images of the nature of science that are specific to 
the laboratory setting, including: (1) the nature of data and 
measurement in empirical work; (2) the nature of investi¬ 
gation in science; (3) the nature of theory in science; (4) the 
nature of explanation in science; and (5) the nature of 
reliable public scientific knowledge. Building on the work of 
Leach (2002), Lederman (2007), and others. Carpi and Egger 
(2010) developed a well-documented and detailed compila¬ 
tion of learning modules that address the question, "What is 
science and how does it work?" that is consistent with and 
builds on the findings of Lederman (2007). Carpi and 
Egger's (2010) scientific process model includes most of the 
points proposed by Lederman (2007), and their "Nature of 
Science" graphic (Fig. 1) provides a succinct visual flow chart 
that can be used not only to teach students about the 
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FIGURE 1: The "Nature of Science" (Carpi and Egger, 2010). Letters A through D were areas of focus within the 
curricular framework in the introduction to curricular redesign. See text for discussion. 
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complexities of science in the real world, but also about the 
excitement and creativity that are part of every scientific 
endeavor. 


TEACHING SCIENTIFIC LITERACY 

Instructors contextualize learning opportunities related 
to the scientific process in a variety of ways (Guerra-Ramos 
et al., 2010), and the efficacy of that learning is affected, in 
part, by pedagogic context (e.g., Ryder et al., 1999; Schwartz 
et al., 2004; Van Eijck et al., 2008). To achieve science literacy 
at the K-12 level, there has been emphasis on learning 
science by doing science for nearly five decades (e.g., Gagne, 
1963; AAAS CSE and Gagne, 1965; O'Neill and Polman, 
2004), and Tobin (1990) supports this view, suggesting that 
students construct their understanding by active participa¬ 
tion, interpretation, and iteration. Notwithstanding the 
efforts of educational institutions at all levels to use these 
techniques, studies indicate that the average American 
remains far from scientifically literate (e.g., Culliton, 1989; 
Alters and Nelson, 2002; Keeter et al., 2007; Miller, 2012), 
suggesting that simply participating in the scientific process 
does not promote scientific literacy. 

Although some researchers suggest that overall learning 
is more effective when knowledge is gained in the context of 
scientific processes (Cashin and Downey, 1995; Smart and 
Ethington, 1995), Lederman (2007) suggests that learning 
goals related to scientific literacy are better achieved by 
making students' participation in the scientific process 
explicit. Further, Schwartz et al. (2004) indicate that teaching 
in the context of hands-on activities requires both explicit 
discussions and opportunities for individual reflection in 
order for students to construct a more complete under¬ 
standing of the process of science. In addition, Jurecki and 
Wander (2012) point out that critical thinking is a key, 
although implicit, component of scientific literacy that is very 
rarely emphasized within scientific curricula, especially at the 
introductory level. This finding is supported by Deng et al. 
(2011), who point out the importance of argumentation, 
which involves making and justifying claims and conclusions 
(Driver et al., 2000; Zohar and Nemet, 2002; Sampson and 
Clark, 2009). 


A NEW SCIENTIFIC LITERACY INITIATIVE 

Supported by a National Science Foundation grant 
(award #0942940) awarded in 2010, faculty within the 
departments of geosciences and chemistry at Trinity 
University collaboratively developed activities related to 
new research-grade instruments and integrated those 
activities into existing course curricula. The new instruments 
funded by the grant included a handheld X-ray fluorescence 
spectrometer (XRF) and an inductively coupled plasma- 
optical emission spectrometer (ICP-OES). The versatility and 
portability of handheld XRF technology permit real-time 
elemental analysis in the laboratory or in the field. The 
handheld XRF unit, approximately the size of a hair dryer, 
produces X-rays that are directed at the sample. When these 
X-rays strike atoms within the sample, many atoms become 
ionized and re-emit X-rays (secondary X-rays) with energies 
that are characteristic of elements present in that sample. 
The secondary X-rays are detected by the handheld XRF, 
permitting identification of those elements. A handheld XRF 


can detect the presence of elements as light as magnesium, 
with many elements detectable at the ppm level. ICP-OES is 
used primarily for the analysis of metals in a liquid. The 
sample is injected into an inductively coupled plasma torch 
as a nebulized spray in order to excite ions and atoms 
present in that sample. The excited species then emit 
electromagnetic radiation characteristic of each species, 
and the intensity of each signal emitted from the sample is 
indicative of each species' concentration. The ICP-OES 
method provides quantitative data, with detection limits in 
the low ppm range. Previous researchers have established a 
link between the use of instrumentation and student 
learning about science (Nakhleh and Krajcik, 1993, 1994; 
Richter-Egger et al., 2010), and these new tools provide 
opportunities for us to better involve students in elemental 
analysis of a range of materials, to delve deeply into how the 
instruments work, how these tools obtain elemental 
information, and how scientists (the students) use this 
information in an explicit scientific process context. 

Our overarching objectives for this initiative included: 
(1) improving student understanding of the scientific process 
and the nature of science; (2) improving student under¬ 
standing of quantitative and qualitative elemental analyses; 
and (3) improving student understanding of the applicability 
of science to real-world problems. To achieve these 
objectives, a significant component of this curricular 
initiative was to provide an explicit, scientific process-based 
framework for learning opportunities in grant-related 
science courses, while also providing students opportunity 
to think critically about the results obtained as a result of 
these processes and to reflect on their participation in 
science. Although both departments have been involved in a 
range of curricular initiatives related to this grant, here we 
focus on the results of our efforts to revise activities within 
an introductory geoscience laboratory course with the goal 
of achieving the objectives listed here. 

STUDY POPULATION AND SETTING 

Trinity University and General Education 
Requirements 

Trinity University is a small (approximately 2,500 
students), private, primarily undergraduate, liberal arts 
college with a predominantly residential student population 
and a semester-based curriculum. Students are required to 
complete general education requirements across a range of 
"understandings," including an "understanding natural 
science and technology" requirement. This requirement 
has the rationale: 

This understanding addresses the need of all students to 
understand the implications and benefits of science and 
technology, along with an appreciation of the potential and 
the limits of science and technology to address societal needs. 
The goal of courses in this category is to promote greater 
literacy in science and technology by teaching students to 
understand: a) the fundamental nature of science; b) the 
methods and results of the natural sciences; c) the 
methodologies of science and technology; and d) the 
relationship between science and technology. (Trinity 
University, 2013, p. 31) 
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FIGURE 2: Study population demographic information for physical geology laboratory course at Trinity University. 


Although not stated explicitly, this understanding 
emphasizes the importance of scientific literacy for all 
graduates. In addition to standard coursework in the natural 
sciences, students are also required to complete a laboratory 
course such as the physical geology laboratory, the course 
that is the focus of our study. 

Introductory Geoscience Laboratories and Lectures 

At Trinity University, students have two primary options 
for fulfilling their general education laboratory requirement 
in geoscience. They may enroll in a studio-based physical 
geology course that integrates lecture, discussion, group 
work, and laboratory activities, or they may enroll in a stand¬ 
alone physical geology laboratory course, which requires 
that students take a separate environmental geology course 
as a pre- or corequisite. Any student who wishes to enroll in 
any of the core courses in the geoscience curriculum must 


complete either the integrated lecture-laboratory course or 
both the environmental geology lecture course and the 
physical geology laboratory course. 

In total, 170 undergraduate students (N = 170) enrolled 
in eight sections of the physical geology laboratory course, 
with two sections offered each semester, including fall 2010, 
spring 2011, fall 2011, spring 2012, and fall 2012. All sections 
were taught by Trinity University faculty and met once per 
week for 3 h throughout a 15 wk semester. The mean 
student age at the time of enrollment was 20.6 y, with a 
standard deviation (SD) of 1.6 y, and 57% of students were 
female. The sample population consisted of 90.1% Cauca¬ 
sian, 12.1% Hispanic, 4.3% nonresident alien/foreign, 2.1% 
black, and 1.4% native American, with 3.6% nonrespondent. 
Students' class varied significantly (Fig. 2), with 13.0% first- 
year students, 37.0% second-year students, 14.8% third-year 
students, 32.6% fourth-year students, and 2.5% fifth-year 
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TABLE I: Summary of curricular changes in physical geology laboratory course. 


Change 

Rationale 

Modified oral and written introductions 
to all laboratory course activities 

To provide better context for each activity within the course and within the scientific 
process framework. 

New reflection questions on many 
activities 

To provide students opportunity to reflect on the ways in which activities relate to the 
scientific process. 

Revised igneous, metamorphic, and 
sedimentary rock laboratory activities 

To provide students a better framework for descriptive and comparative research. To 
provide students a more realistic scientific experience, especially in regard to 
interpretation of the environments of rock formation. 

Added course reader 

To provide a single source for all laboratory support materials that is organized within a 
scientific process framework. 

New prelaboratory quizzes 

To provide incentive to complete prelaboratory readings prior to activities and to increase 
the time available for faculty-student interaction when students are actively involved in 
research. 

New scientific method activity 

To spur students to think carefully about their own conceptions of the scientific method, 
prior to any class discussion of the scientific process. To provide a model and a resource 
for the scientific process 1 as practiced in the real world. 

New XRF elemental analysis activity 

To introduce students to atomic structure, the power of the periodic table, and X-ray 
instrumentation and elemental analysis. To demonstrate the importance of similar tools to 
the advancement of knowledge within the scientific process. 

New "real-world" investigation of 
campus soils, bedrock, and human 
influence 

To further demonstrate the power of elemental analysis in answering research questions 
that can be applied to real-world challenges. To involve students in research that involves 
research question construction, hypothesis formation, hypothesis testing, and hypothesis 
modification/rejection. 


1 Primary resource: Carpi and Egger (2010). 


students. The mean grade point average of students upon 
entering the course was 3.26 (on a 4 point scale), with a SD 
of 0.45. On course evaluations, students reported that 87.1% 
took the course to fulfill general education requirements, 
17.8% to fulfill requirements for their major, 5.5% to fulfill 
requirements for their minor, 1.2% to prepare them for 
postgraduation plans, and 3.1% to fulfill elective require¬ 
ments. Students' reasons for registering in the course most 
commonly included course content, the instructor, and 
availability at registration, and 91.9% of students perceived 
the course as moderately to substantially difficult, with 3.7% 
considering the course slightly difficult and 4.4% considering 
the course extremely difficult. The level of effort expended, 
according to students, was primarily moderate to substantial, 
at 90.0%, with 5.6% considering their effort slight and 4.4% 
considering their effort extreme (Fig. 2). 

METHODS: CURRICULAR REVISION 

Geosciences faculty developed a revised framework for 
the department's introductory-level physical geology labo¬ 
ratory beginning in the summer of 2010. Starting with the 
student learning goals for the NSF-funded initiative, we 
worked to integrate ideas and activities associated with 
scientific literacy throughout both existing and new labora¬ 
tory activities to achieve those goals. Coupled with a new 
focus on the scientific process, we also worked to integrate 
use of the two new research-grade instruments (the 
handheld XRF and ICP-OES) into the laboratory's curricu¬ 
lum. All changes were instituted during the 2010-2011 
academic year. 

Prior to the fall of 2010, physical geology laboratory was 
taught as a series of activities that were designed to 
introduce students to laboratory methods and to provide 


students opportunities to improve their knowledge of the 
solid Earth and Earth processes. At the beginning of every 
course meeting, faculty would briefly aid students in making 
important connections between earlier activities and the 
student activity to be performed that day. Faculty supplied 
materials specific to the day's activity, which included both 
guidelines for the activity itself and supporting documents to 
aid them in completing the activity (e.g., igneous rock 
classification chart and definitions of textural and composi¬ 
tional terms). We recognized that students who completed 
the course had achieved the primary learning goals at some 
level, but it was not clear to faculty that students understood 
how these activities related to the process of science, 
especially the way in which their participation in these 
activities could be related to the work that scientists 
undertake or the manner in which science impacts society. 
In addition, none of these activities involved the use of 
qualitative or quantitative elemental analysis to answer 
research questions, something that is vitally important to 
research across all scientific disciplines. 

In order to achieve initiative objectives and to improve 
overall student learning of the fundamental skills and 
concepts of physical geology, faculty instituted a number of 
important changes to the curricular structure of the physical 
geology laboratory (Table I). These changes were integrated 
into the course at a number of different levels, from relatively 
simple, minor modifications to the addition of a course 
reader and new activities. 

Providing New Context: A Visual Model for the Nature 
of Science 

To combat many of the flawed conceptions about the 
nature of science and the scientific process, we decided to 
use a visual model that reveals the robust, dynamic, and 
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TABLE IIA: Chart to be completed by student for igneous rocks activity (prior to revision). 


Sample 

Texture 

Minerals Identified 

Overall Composition 

Rock Name 







diverse nature of science as practiced in the real world. We 
built much of our framework around Carpi and Egger's 
(2010) scientific process model, focusing especially on the 
"Nature of Science" graphic (Fig. 1), which provides 
students with a succinct visual flow chart that can be used 
not only to teach students about the complexities of science 
in the real world, but also about the excitement and 
creativity that are part of every scientific endeavor. 

Although we guided class discussion about many parts 
of the scientific process and the nature of science at many 
times throughout the course, we focused especially on 
several components of the scientific process represented in 
Fig. 1. These included the "valve" that constrains research 
questions (A in Fig. 1), research methods used by scientists 
(B in Fig. 1), the ways in which analysis of data can be used 
to develop interpretations and testable hypotheses (C in Fig. 
1), and the "valve" that constrains how scientists dissem¬ 
inate ideas that result from their research (D in Fig. 1). We 
also discussed the importance of theories and laws in 
science, thinking especially about how these terms are 
understood by the general public relative to how these terms 
are actually used by scientists. We purposefully integrated 
many of the modules from Carpi and Egger's (2010) work 
into course materials, which ranged from a new laboratory 
reader to class discussions of how individual activities might 
fit within a scientific process framework. 

Revision of Existing Laboratory Activities 

By modifying both the oral and printed introduction to 
laboratory activities, we provided students with a more 
succinct picture of how each activity fit within both earlier 
and later physical geology concepts, as well as, in many 
cases, how a given activity might fit within the scientific 
process (Fig. 1). We also added reflection questions on many 
activities that provided students with an opportunity to 
reflect and demonstrate their overall understanding of both 
how that activity fit within the scientific process framework 
and how material learned could be related to knowledge 
gained in past activities. Finally, we significantly revised 
igneous, sedimentary, and metamorphic rock activities to 
better reflect the way that a geologist would observe and 
interpret the rocks, guiding students in a way that 
emphasized the importance of descriptive and comparative 
research methods (B in Fig. 1) to the overall understanding 
of geologic processes. 

TABLE IIB: Revised chart format for igneous rocks activity. 

Rock Sample #3 

Texture: 

Mineral observations (box compressed due to space 

constraints): 

Overall composition: 

Interpretation (box compressed due to space constraints): 

Possible plate tectonic environment: 


For example, when students begin working with igneous 
rocks, we introduce them to the primary criteria that are the 
basis for the igneous rock classification system. Following an 
instructor-led discussion of the importance of texture and 
composition in the interpretation of igneous rocks, students 
begin observing and describing a set of samples. Prior to the 
revision of this activity, students were given the classification 
charts that would aid them in naming the rock based on 
texture and composition (e.g., porphyritic basalt), and 
students were supplied with a standard test kit, (hand lens, 
nail, glass, and streak plate), a binocular stereoscope to view 
hand samples, and a visual percent chart to estimate mineral 
percentages in rock samples. With these tools and support 
materials, students examined hand samples and completed a 
chart with the columns shown in Table IIA. 

Based on laboratory experiences, faculty felt this was a 
flawed approach. Instead of using careful observation and 
description to classify igneous rocks, students would 
commonly pick up a sample, guess what rock it was, and 
use the classification chart to fill in the minerals that were 
expected to be present. While the best students spent time 
identifying visible minerals in the rock, with 14 total samples 
to describe and classify, even those students focused more 
on finishing the laboratory exercise than on understanding 
the importance of the rock classification system or using 
their descriptions to better understand the environment of 
formation for a given rock. Thus, we revised the activity to 
emphasize the importance of careful observation and to 
engage students in the thought processes that enable 
geologists to interpret environments of rock formation. 

We reduced the total samples to be described from 14 to 
five, and we changed the format of the laboratory handout to 
emphasize those items we felt to be most important (Table 
IIB). The new chart format included a significant amount of 
space for students to describe visible minerals and/or 
groundmass characteristics, including the average measured 
size and range in size of mineral crystals, the color of the 
mineral in the sample, and the approximate percentage of 
the mineral present. The chart also included a new 
"Interpretation" box, where students were required to 
interpret the environment of formation for each rock, using 
the evidence that they presented in their "Texture" (i.e., 
phaneritic, aphanitic, or porphyritic aphanitic), "Mineral 
observations," and "Overall composition" (i.e., mafic, 
intermediate, or felsic) boxes. This interpretation included 
the location(s) where these rocks most likely formed (e.g., 
magma chamber), the magma composition present, and the 
rates of cooling expected to generate a rock with the 
composition and appearance of their hand sample. Finally, 
students were required to include at least one tectonic 
environment where such a rock could form. Importantly, we 
did not supply students with either a classification chart or 
an opportunity to name the rock (compare Figs. 2A and 2B). 
We included discussion and supporting materials about the 
classification system and rock-naming conventions as part of 
the next week's activity. We also revised other activities that 
involved hand samples, especially those focused on sedi- 
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mentary and metamorphic rocks, for similar reasons and in a 
similar way. 

New Course Reader 

Prior to this initiative, geoscience faculty distributed 
supporting materials prior to each activity, discussing the 
importance of those materials to the concepts and skills 
introduced that day. While the support materials were 
important references for students working on laboratory- 
related materials, the relationships between supplemental 
materials either to each other or to the overall course were 
not explicit. To improve both the quality of the supplemental 
materials and the relationship between the materials and 
overall course goals, we combined all supplemental mate¬ 
rials into a single course reader, while also providing 
additional text to introduce the reader to materials and to 
provide explicit reminders of the research methods being 
used in completing each activity. We also added a significant 
introduction to the reader, focusing on the scientific process, 
using Carpi and Egger (2010) as a primary resource. Near the 
end of the reader, we included all classification diagrams and 
other materials that students would use frequently (e.g., 
periodic table, geologic timescale, a graphic to aid in percent 
estimation). 

With the introduction of a new course reader, we added 
a new set of expectations for student preparation. Prior to 
each laboratory, students were expected to read the section 
of the course reader related to activities performed that day. 
To provide incentive for preparation, we distributed prel¬ 
aboratory quizzes at the beginning of each meeting. We 
hoped these quizzes would also provide valuable feedback 
about which topics were most challenging for students. 

New Activities 

We were limited in the number and type of activities 
that we could add to the physical geology laboratory 
curriculum. Therefore, most of our new activities were 
relatively short, focused activities designed to engage 
students in both the content and skills necessary to achieve 
learning goals. 

We developed a simple activity to engage student in 
active reflection about their conceptions of the scientific 
method/process. This activity was designed to spur students 
to think about how their own ideas about scientists and the 
nature of science meshed with the way that science is 
practiced in the real world. Each student worked with a 
partner to develop a flow chart that was a graphical 
representation of their views about science. This exercise 
was used as a springboard for a classwide discussion of the 
scientific process and a general introduction to the course. 

We added a new activity to introduce the handheld XRF 
and elemental analysis to students. The activity included an 
introduction to the physics behind elemental analysis and a 
discussion about atomic structure and the periodic table. 
This elemental analysis activity immediately followed an 
activity focused on the exploration of minerals and mineral 
properties, so the first part of the exploration included 
analysis of powdered mineral samples, using two samples of 
the same mineral with different colors (e.g., Ca-rich 
plagioclase feldspar versus Na-rich plagioclase feldspar). 
The activity was designed to encourage students' recognition 
of the power of research-grade instruments to reveal 
detailed elemental information. In addition, another learning 


goal was for students to learn that color alone was not a 
good diagnostic for mineral identification; variations in 
mineral color are controlled by slight changes in the 
elemental composition of minerals. Finally, we guided 
discussion to include the definition of a mineral, thinking 
carefully about how elements substitute into positions 
within a mineral's crystal structure. 

Finally, we included a multistep research project that 
involved the use of both the handheld XRF and the ICP- 
OES. This research project was developed as an analysis of 
bedrock and soils on the Trinity University campus, focusing 
especially on the composition of soil horizons relative to 
both the regolith and bedrock immediately beneath much of 
the campus as well as to human activities. The project steps, 
spread throughout the course as short exercises, included: 
(1) a guided research question development exercise, based 
on detailed student observations of the sampling locations 
and background information about campus construction 
provided by instructors; (2) XRF analysis of the soils and 
bedrock samples to provide information about the relative 
elemental concentrations in the soils and bedrock; (3) a 
hypothesis formation exercise, based on the originally posed 
research questions and the elemental data collected by XRF; 
(4) ICP-OES analysis of the same samples to provide 
quantitative data about elements chosen by students in the 
third step of the project; and (5) a closing exercise, where 
students use all available data, their original research 
questions, and their hypotheses to develop new hypotheses 
that best explain the data. This final project also includes a 
section that pushes students to think carefully about what 
other information would help them in their investigation of 
soils on campus. 

METHODS: ASSESSMENT FRAMEWORK 

Students' ideas about the nature of science are so 
dynamic and varied that reliable, valid assessment of student 
learning in topics related to the nature of science is difficult 
(Lederman et al., 1998). In addition, students' ideas about 
the nature of science might appear to be different depending 
on the questions asked, the assessment method used, and 
when an assessment tool is used, especially considering the 
blend of experiences through which students construct their 
ideas (Leach et al., 2000; Roth and Lee, 2007; Van Eijck et al., 
2008). In order to most effectively measure student learning 
related to our overarching initiative objectives, we designed 
a wide range of both quantitative and qualitative assessment 
tools that we administered at different times throughout the 
course (Table III) and that were analyzed by multiple 
observers. For all qualitative assessment tools, we utilized 
standard qualitative content analysis, which consists of 
"identifying, coding, categorizing, and labeling primary 
patterns" (Patton, 2002) in our analysis of that data. Our 
assessment framework thus utilizes both methods and 
analyst triangulation, as defined by Patton (2002), to 
evaluate the consistency of our findings. With this design, 
we attempted to avoid the bias that is common in single¬ 
method, single-observer, and/or single theory studies 
(Denzin, 1989). 

The assessment tools were structured to provide 
frequent feedback to both students and instructor, which 
could then be used to both inform activities performed later 
in the course and in subsequent offerings of the physical 


252 Surpless et al. 


J. Geosci. Educ. 62, 244-263 (2014) 


TABLE III: Chronological list of assessment tools and response types for physical geology laboratory. 


Week 

Assessment Tool 

Response Type 

Objective(s) 

Assessed 1 

1 

Prelearning questionnaire 2 

Likert scale questions; free response 

1, 2, 3 

1 

Scientific method construction 2 

Free response, including flow chart construction 

1 

3 

Trinity rock/soil project: introduction 

Free response (research question formulation) 

1, 2, 3 

4 

Trinity rock/soil project: XRF analysis of 
minerals, rock, and soil samples 

Free response (laboratory activity and follow-up summary) 

1, 2, 3 

5 

Quiz (elemental analysis) 

Multiple-choice and free response 

2 

6 

Practical exam I 

Free response (observation and interpretation of rock 
samples) 

1, 2, 3 

7 

Rock interpretation and the practice of 
science in geology 

Free response (self-reflection on students' use of the 
scientific process in completing the laboratory activity) 

1, 3 

9 

Trinity rock/soil project: ICP-OES analysis 

Free response (laboratory activity) 

1, 2, 3 

11 

Quiz (scientific process) 3 

Multiple-choice and free response 

1 

13 

Practical exam II 3 

Free response (both rock interpretation and scientific 
process) 

1, 3 

15 

Postlearning questionnaire 3 

Likert scale questions; free response 

1, 2, 3 

N/A 

NSF summative report 3 

External evaluator used all available data (see above) in 
addition to other methods of data collection (see text for 
discussion) 

1, 2, 3 


'Objectives listed in "A New Scientific Literacy Initiative" section. 
2 Prelearning activities addressed in text discussion. 

3 Postlearning activities addressed in text discussion. 


geology laboratory. Importantly, we included assessment of 
student learning and overall project efficacy by an external 
evaluator (Table III). Many assessment tools, including 
student questionnaires and faculty interviews, were designed 
to measure the same constructs in both pre- and postlearn¬ 
ing applications. However, in several cases, time constraints 
prevented us from designing a prelearning assessment tool 
that could be directly tied to a postlearning assessment tool. 

Questionnaires (Table III) included both Likert scale 
statements as well as open-ended free-response questions to 
elicit deeper information about student learning. We 
included both pre- and postlearning questionnaires, which 
are particularly effective at assessing the attainment of 
learning goals (Libarkin, 2001). In addition to assessing the 
attainment of initiative goals, the prelearning questionnaire 
permitted us to tailor students' learning opportunities to best 
match existing student attitudes and conceptions (Libarkin, 
2001) and maximize the efficacy of student learning 
(Hofstein and Lunetta, 2003). We also used assessment 
questions that target student attitudes toward science and 
the scientific method (Aikenhead and Ryan, 1992; Bradford 
et al., 1995; Libarkin, 2001), and we utilized Carpi and 
Egger's (2010) discussion of data analysis and interpretation 
to help mold the Likert scale statements. 

We embedded assessment tools in laboratory activities, 
quizzes, and practical exams throughout the term (Table III). 
Two new quizzes and revised practical exams (Table III) were 
designed to assess student learning goals. With this overall 
structure, we provided students frequent opportunities for 
both active participation in scientific investigations and 
interpretation within an explicit scientific process frame¬ 
work. This assessment plan also provided us with frequent 
feedback about student learning, so that we could tailor 
subsequent activities to mitigate problems that we observed 


in either the activities themselves or in the context that we 
provided for those activities. It is important to note that 
instructors commonly provided oral feedback during class 
discussions that took many forms, including, but not limited 
to: correcting misconceptions as students expressed them; 
providing additional context for a discussion topic (if deemed 
necessary and relevant); pointing out connections between a 
given activity and prior activities; and pointing out the 
relation of the day's activities to the scientific process. We 
provide all course materials related to this assessment effort 
in the Supplemental Materials (available at: http://dx.doi.org/ 
10.5408/13-073sl and http://dx.doi.org/10.5408/13-073s2). 

External Evaluation Methods 

Our external evaluator (coauthor M. Halx) provided 
both quantitative and qualitative evaluation of grant-related 
activities (Table III). The quantitative methods consisted 
primarily of Likert scale questions on pre- and postlearning 
questionnaires, as described earlier. The qualitative methods 
included constant comparative coding and analysis of both 
the student text responses to open-ended questions on the 
pre-and postlearning questionnaires and of transcripts from 
faculty, administrator, and staff interviews. Although similar 
to standard comparative coding (discussed earlier herein), 
the constant comparative method includes the addition of a 
dynamic process in which the researcher constantly revisits 
and compares previously gathered data to determine if 
adjustments to the research process are necessary. This 
method encourages rigor and researcher accountability 
through open, axial, and selective coding (Strauss and 
Corbin, 1998). This three-tiered process requires multiple 
passes through the data to assure that data saturation has 
been achieved and that no themes have been overlooked. In 
addition, Halx tested the validity of his findings by verifying 
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researcher interpretations with evaluation participants, a 
process also known as "member checking" (Patton, 2002). 
"Analytic memos" (Saldana, 2009) were also maintained 
throughout the evaluation period. These "memos" are 
frequent notes of any analytic or methodological thought 
or research question recorded as those thoughts occur, 
which are later used to inform the analysis process (Saldana, 
2009). During the analysis process, no automated discourse 
analysis program was used. Instead, in vivo coding (Saldana, 
2009) was utilized. This coding is drawn from the 
participants' own words and avoids researcher-assigned 
labels (Saldana, 2009). This method provides direct individ¬ 
ual insights that are unaffected by coding processes. 
Separate in vivo code grouping schemes were developed 
for faculty, administrators, staff, and students in order to 
parallel the specific research pursuit for each participant 
pool. For the faculty, administrators, and staff participants, 
the coding scheme focused on the value of the new 
instruments and effectiveness of the curricular changes. For 
the student participants, the coding schemes focused on 
evidence of student learning and increased understanding. 
The response rate to the student questionnaires averaged 
approximately 90% for each semester measured (N = 170 for 
the entire study). We summarize the external evaluator's 
summative evaluation in the following section. 

RESULTS: PRE- AND POSTLEARNING 
ASSESSMENT DATA 

Although we developed a detailed assessment frame¬ 
work with frequent feedback mechanisms for activities 
throughout physical geology laboratory, we here focus on 
the total change in student skills and knowledge, focusing 
especially on student assessment data from the beginning 
and the end of the course. For prelearning assessment, we 
focus on the prelearning questionnaire, the scientific method 
construction activity, and the Trinity rock/soil project 
introduction (Table III). For postlearning assessment, we 
focus on the postlearning questionnaire, the scientific 
process quiz, the second practical exam, and the last step 
in the Trinity rock/soil project ("Trinity rock/soil project: 
ICP-OES analysis" in Table III). For the class activities, we 
use both the questions/statements posed as well as excerpts 
from student responses (original student responses are 
included in the Supplemental Materials). Finally, we 
summarize the student learning section of the summative 
evaluation provided by the project's external evaluator. 

Prelearning Questionnaire 

Students responded to 4 Likert scale statements (Fig. 3), 
with a scale that included: strongly disagree (SD; 1 point); 
disagree (D; 2 points); neutral (3 points); agree (A; 4 points); 
and strongly agree (SA; 5 points). In responding to the 
statement, "I understand the scientific method," 85.6% of 
students selected agree or strongly agree, with a standard 
deviation of 0.71 from the average of 4.04 (Fig. 3; all 
prelearning responses in black). Students responses to the 
statement, "I know the basic structure of the atom," 
suggested slightly less student confidence in understanding, 
with an average of 3.79 and a standard deviation of 0.90, but 
74.3% of students did select agree or strongly agree (Fig. 3). 
For the statement, "I understand what an X-ray is," 62.8% of 
students selected agree or strongly agree, with an average of 


3.51 and a standard deviation of 0.88 (Fig. 3), indicating still 
less confidence in their understanding of that concept, 
relative to the first two statements. Finally, the statement, "I 
understand the basic relationship between electron shells 
and energy," elicited only 38.5% agree or strongly agree 
responses, with an average of 3.10 and a relatively large 
standard deviation of 1.04 (Fig. 3). These data suggest that 
prior to taking the physical geology laboratory course 
students were most confident about their understanding of 
the scientific method, were less confident about their 
understanding of atomic structure or X-rays, and were the 
least confident about electron shell-energy relationships. 

Students also responded to four other statements, with 
possible responses including agree, disagree, and cannot 
decide (Fig. 4). In addition, we provided students with the 
opportunity to explain why they chose the response that 
they did. Figure 4 includes bar graphs that display student 
responses (black bars = prelearning; gray bars = postlearn¬ 
ing), and to the right of the graphs are selected student 
responses. In responding to the statement, "Good scientific 
data can only be interpreted in one way," 80% of students 
disagreed, with only 5.1% agreeing with the statement. Also, 
98.7% of students agreed with the statement, "Even when 
scientific investigations are done correctly, the information 
that scientists discover may change in the future." Students 
responding to the statement, "Scientific research must 
involve experimentation," selected agree 66.6% of the time, 
while 23.1% selected disagree, and 10.3% could not decide. 
Finally, 41.1% of students agreed with the statement, "If 
multiple hypotheses can explain the same set of observa¬ 
tions, the simpler hypothesis is preferred by scientists," 
while 29.4% disagreed, and 29.5% could not decide. We also 
requested that students select all fields (e.g., biology, 
chemistry, geology) for which they thought X-ray fluores¬ 
cence analysis could be applied to answer research questions 
(Fig. 4). 

Scientific Method Construction Activity (Prelearning) 

In order to assess student understanding of the scientific 
process, we requested that students respond to the following 
free response opportunity: Explain the scientific method, 
using a flow chart to aid in your explanation. Students 
worked with a partner to develop their explanation and flow 
chart. Student explanations of some parts of the scientific 
method were surprisingly consistent, with all student flow 
charts including hypothesis formulation, experimentation, 
and conclusions. Nearly all explanations included either a 
linear or a circular flow chart. Two samples of the circular 
models are included in Fig. 5, and four samples of the linear 
models are shown in Fig. 6 (raw scans of these diagrams are 
included in the Supplemental Materials). 

These flow charts all start with an observation, a 
problem, or a question (Figs. 5 and 6). In two cases, (Figs. 
5B and 6A), students included research prior to hypothesis 
formulation. In all cases, students included hypothesis 
formulation and followed that step with experimentation 
(or hypothesis testing). With two exceptions (charts in Figs. 
6C and 6D), students include data collection as a step that 
follows experimentation/hypothesis testing, and in these 
same scientific method models, students also include data 
analysis after data collection. In three of these four student 
flow charts (Figs. 5A, 5B, and 6A), data analysis was followed 
by a conclusion, but in the fourth of these flow charts (Fig. 
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FIGURE 3: Likert scale pre- and postlearning student-response data. In graphs, black bars represent prelearning 
responses, and gray bars represent postlearning responses. 


6B), students also included peer review and check hypothesis 
as two additional steps prior to the conclusion. In the case of 
the circular flow charts (Fig. 5), conclusion formulation leads 
back to the "initial" step. 

Most student teams did not include the development of 
theories or laws in their responses to this free-response 
opportunity. Those who did include theory development as 
part of the "conclusion" stage suggest that once a hypothesis 
is proven reproducible multiple times, it becomes a theory. 


During subsequent student discussion of the terminology 
used by scientists (during the same laboratory meeting 
period), it became apparent that students considered a 
"proven" hypothesis (i.e., reproducible by multiple other 
scientists) to be a theory. Most students also considered 
scientific theories a matter of opinion, since they had not 
been proven true. Importantly, students considered a theory 
that had been proven true to be a law, and a law was an 
absolute fact that could not be questioned. We believe that 
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Example student responses 


Pre-learning 


"If a hypothesis is strong and 
unfalsifiable, then yes." 

"Data does not give answers; 
theories and logical conclusions do." 
"Data may reflect certain 
experimental conditions." 
"There are several ways to interpret 
data." 

"There are different hypotheses and 
ways to test data." 

"It can be interpreted in many ways." 


"Science is always evolving." 
"Improvements in technology may 
lead to more accurate or complex 
data collection." 

"Theories can always change and 
new data can change old 
understandings." 

"New information leads to different 
ways of thinking about an issue." 
"There are many innovations that 
change the future." 


"Part of the scientific method - it 
can't be argued with." 
Something must be tested to be 
proven." 

"How do we find out anything 
without first testing it?" 
"Everything can undergo 
experimentation, under controlled 
circumstances." 

"It serves as evidence to the 
hypothesis." 


"I do not know enough 
about...scientists to know how 
scientists respond..." 

"Keep it simple, easier to 
understand." 

"The simplest hypothesis may be 
the most convenient, but it may not 
include all...factors." 
"Occam's razor." 

"Not necessarily the simpler but the 
better." 


Post-learning 

"Data is just the results of an 
experiment. What it means is left 
for the scientist to figure out, and it 
is possible that it can support more 
than one conclusion." 
"Scientific data can be interpreted 
in various different ways; it is not 
tailored to one explanation." 
"Answers and experiments may 
have a single outcome, but they 
may mean different things to 
different fields." 

"Technology changes, so our ability 
to get more accurate results means 
that sometimes our best informa¬ 
tion changes." 

"New discoveries in science can 
change the way we view data." 
"Our knowledge is limited, so in the 
future when we make new 
discoveries, we will change the 
"inputs" and get different "outputs." 
"Science is subjective, and new 
"findings are always in progress." 


"Not necessarily. It can involve 
observation and description..." 
"Experimenting is how researchers 
get their results/conclusions." 
"Observation is possible." 
"Some sort of experimentation, but 
not necessarily a formal 
experiment." 

"There are other methods besides 
experimentation." 

"You can use...comparison or 
modeling." 


"Occam's razor." 

"Simpler is always better." 
"Why choose more complex?" 

"It depends on the hypothesis, but 
sometimes the simpler one is 
better." 

"This is a rule in science." 
"Multiple hypotheses require more 
explanations and would require 
specific things to be observed in 
order for the hypothesis to be true." 
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FIGURE 4: Pre- and postlearning student-response data. In graphs, black bars represent prelearning responses, and 
gray bars represent postlearning responses. 
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FIGURE 5: Two samples of student-authored circular-style scientific process flow charts. 


the omission of theory and law from most student flow 
charts was influenced by class discussion that led in to the 
activity. 

Trinity Rock/Soil Project Introduction (Prelearning) 

As described in the "Curricular Revision" section, we 
developed a new interdisciplinary project about soil devel¬ 
opment on the Trinity University campus. This activity was 
intended to be an introduction to the process of science as 
practiced in the real world. To provide context, we focused 
on Fig. 1, spending significant time discussing the ideas near 
"A" on the graphic, so that subsequent student observations 
and question formulation could be placed within the 


scientific process framework. After students were introduced 
to the framework for the multistage project, students were 
required to make observations and formulate research 
questions that could be answered by elemental analysis 
methods of the soils and bedrock samples from one location. 

Each student worked with a partner to author a 
minimum of five possible research questions. These ques¬ 
tions (sample student questions included in Table IV) varied 
widely in their content and revealed a general lack of 
understanding both about elemental analysis and about the 
formulation of good research questions. However, during a 
subsequent activity (week 4 activity; Table III), instructors 
chose the best of the student-authored research questions to 
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FIGURE 6: Four samples of student-authored linear-style scientific process flow charts. 
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TABLE IV: Sample prelearning student research questions for 
the Trinity soil/bedrock project. 

What fill was used prior to developing the campus? 

What is the age difference between the layers? 

What is the difference in mineral composition of the layers? 

What are the different thicknesses of the layer's? 

What period is the bedrock from? 

How many times has construction affected the soil? 

How is the soil closer to the bedrock different compared to soil at the 
top? 

Does the location affect the composition of samples? 

Is there a higher rock concentration closer to the bedrock? 

Why does the soil start on top of the bedrock? 

What did the area look like 50 years ago? 


lead students to develop good, testable hypotheses, which 
were then used to guide the later ICP-OES analysis activity 
(Table III; week 9 postlearning activity). 

Trinity Soil/Rock Project: ICP-OES Analysis 
(Postlearning) 

In an earlier activity, students obtained elemental data 
using XRF analysis of the soils and rocks (week 4 activity; 
Table III). Although qualitative, this analysis did provide 


students information about the relative concentrations of 
elements in the rock and soil samples analyzed. Based on 
these data, students developed hypotheses that they thought 
could be tested with quantitative analysis by ICP-OES 
analysis (see four examples of student-authored hypotheses 
from week 4 activity in Table V). With this elemental 
information, students in a linked chemistry course analyzed 
eight elements present in the soils and bedrock (Mn, Sr, Ti, 
Zn, Ni, Cu, Fe, and Ca) by ICP-OES. The elemental 
concentrations from these analyses were then provided to 
students in the physical geology laboratory course for this 
week 9 activity). 

In this week 9 activity (Table III), students in the physical 
geology laboratory analyzed the ICP-OES data in order to 
evaluate initial hypotheses using those data, come to some 
initial tentative conclusions, and develop new hypotheses 
based on both the original XRF data and the new ICP-OES 
data (Table V). Each student worked with a partner to use 
the data provided to answer the questions posed. Finally, 
students thought about how these newly developed 
hypotheses might be tested, assuming unlimited resources. 
We include sample student responses in Table V. 

Scientific Process Quiz (Postlearning) 

We designed this brief assessment tool to help us 
determine whether students still held many of the common 
misconceptions that we observed earlier in the course. 
Although the quiz also included questions about absolute 


TABLE V: Student-authored hypotheses, questions, and sample student responses from ICP-OES analysis activity (week 9 activity 
in Table III). 


Student-authored hypotheses to be evaluated by students: 

A) The soil is derived from the limestone. 

B) The soil is not derived from the limestone. 

C) The soil samples closer to the surface have higher concentrations of most elements. 

D) The soil has a lower concentration of Ca than the limestone. 

1. Describe any trends or patterns you see in the data (students were given elemental data in ppm). 

"Zn, Ni, and Cu are the least abundant elements throughout the soil layers." "Ti increases in abundance with depth." "Cu decreases as it 
approaches bedrock." “More Zn as it got closer to the surface." "Less Ni in [sample] B, with no data for C or D." 

2. Are these data consistent with any of the hypotheses presented (see four student-authored hypotheses, above)? Do these data disprove 
any hypothesis? Explain your answers. 

See student-authored hypotheses and responses in Supplemental Materials. 

3. Given these new data, what interpretations can you make about the relationships among bedrock, soil, and human activities? 

"The bedrock is the oldest and most accurate example of the original composition of the area. Based on the relatively close relationship of 
amounts of particular elements between samples C and D, it can be implied [inferred] that the composition of the bedrock helps to determine the 
composition of the overlying soil. At the same time, human activity has definitely affected the upper layers of soil as seen by the higher amounts 
of certain elements that are not in abundance in the bedrock." 

"There is not really a clear trend. Some [elements] increase as they get closer to the bedrock while some decrease. Since that is the case so soil 
might be more affected by the bedrock while certain [samples] are more affected by human activities as evidenced by [elemental] concentrations." 

4. Develop a new hypothesis that is consistent with all observations and data. You can either modify one of the existing hypotheses or 
write a new one. 

"Human activity has altered the elemental (and chemical) composition of the soil." "Human activity has altered the top 2 layers of soil." 

"The Ca in the soil was derived from the limestone, whereas the other elements were derived from human interactions with the soil." "There is 
something besides depth that affects [elemental] concentrations in soil because the [elemental] concentration var(ies) so much with each sample. 

So human activity might be a cause of this variation." 

5. How could you test your new hypothesis with quantitative analysis, assuming unlimited resources? 

"Have two soil samples; one that is affected by humans, one that is not and see how they are different." "You could use the XRF to analyze 
the elemental composition of fertilizer used at Trinity to see if it matches the elements found in the soil." 
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TABLE VI: Questions from postlearning scientific process quiz. 


Uniformitarianism is: 

a. the principle that best explains why all sedimentary rocks are deposited at the same rate. 

b. the opposite of catastrophism. 

c. the longest word in the English language. 

d. best phrased, "the present is the key to the past." 

e. both b and c. 

A theory is: 

a. a tenuous explanation for a specific observation. 

b. an explanation for natural phenomena that is logical, testable, and predictive. 

c. an educated guess by a scientist. 

d. the next step in the process of science, after a hypothesis is developed. 

e. not well supported by evidence; only when enough evidence is gathered will a theory become a law. 

Research methods: 

a. include experimentation, modeling, observation, and comparison. 

b. are utilized almost exclusively in laboratory settings. 

c. are virtually identical from scientific discipline to scientific discipline. 

d. have remained largely the same over the past 150 years. 

e. usually involve bubbling vials, with occasional explosions. 

True/False statements: 

A theory cannot become a law. 

The important questions that are investigated by scientists are not affected by government funding priorities. 

Since nearly all life that has existed in the past is well-preserved in the rock record, we should be able to completely construct the evolution of all 
life, if given enough time. 

A hypothesis becomes a theory once that hypothesis is supported and published in the scientific literature. 


age dating, we used these questions (Table VI) to help guide 
a class discussion about the scientific process, which 
immediately followed the quiz. Students performed remark¬ 
ably well on the quiz, scoring an average of 8 out of 9 points, 
and subsequent student discussion revealed that students 
better recognized the complexity of the scientific process, as 
practiced in the real world. In fact, students commonly 
referred directly to the scientific process graphic of Carpi and 
Egger (2010) (Fig. 1) to help make their points. 

Practical Exam 2 (Postlearning) 

We authored three questions on the second practical 
exam in order to measure student learning about different 
aspects of the process of science. These questions included: 
(1) Give at least two research methods that you have 
employed during this course. Explain. (2) Explain why a 
theory can never become a law, and give one example of a 
theory (in geology) and one example of a law (in geology). 
(3) What is peer review, and why is it important to the 
process of science? 

We hoped to assess whether students had learned a 
more accurate conception of the scientific process, as 
summarized by Carpi and Egger (2010) (Fig. 1). Importantly, 
we designed these questions to evaluate specific areas where 
we had observed common misconceptions about the 
scientific process in prelearning activities (especially at 
points A-D on Fig. 1). Since we addressed these concepts 
using different pedagogical methods throughout the course 
(Tables III and IV)/ we thought these questions represented a 


fair way to both assess the effectiveness of our curricular 
redesign as well as to provide students opportunity to 
demonstrate their understanding of the scientific process. 
Sample student responses are included in the Supplemental 
Materials. 

Postlearning Questionnaire 

Students responded to the same Likert scale question¬ 
naire used for prelearning assessment (see earlier herein). In 
responding to the statement, "I understand the scientific 
method," 98.2% of students selected agree or strongly agree, 
with a standard deviation of 0.51 from the average of 4.33 
(Fig. 3; postlearning responses in gray). Students' responses 
to the statement, "I know the basic structure of the atom," 
suggested slightly less student confidence in understanding, 
as in the prelearning questionnaire, with an average of 4.17 
and a standard deviation of 0.64, but 84.3% of students did 
select agree or strongly agree (Fig. 3). For the statement, "I 
understand what an X-ray is," 92.6% of students selected 
agree or strongly agree, with an average of 4.07 and a 
standard deviation of 0.54 (Fig. 3). Finally, student responses 
to the statement, "I understand the basic relationship 
between electron shells and energy," elicited only 90.8% 
agree or strongly agree responses, with an average of 4.09 
and a standard deviation of 0.59 (Fig. 3). 

As with the prelearning questionnaire, students re¬ 
sponded to five statements related to the scientific process 
and elemental analysis (Fig. 4; postlearning responses in 
gray). In responding to the statement, "Good scientific data 
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TABLE VII: Student responses related to XRF and ICP-OES 
use, as summarized by coauthor Halx. 

"It was interesting seeing the graphs display what was happening on 
such a small level ." 

"I like the live-action results and the difference between soil in one 
place and soil in another place that was not very far away." 

"Who knew there was so much detail in a rock?" 

"It was very interesting and exciting when we got to use the 
(instrument)." 

"It was a lot of fun. We studied what elements were in 
various.. .materials." 

"The XRF measured the elemental composition of (soil).. .which got 
me to think that dirt is much more interesting than I thought." 

“We used the XRF to analyze a necklace and saw all of the elements 
in it.. .it was so cool!" 

"The XRF gives us qualitative measure, while the ICP gives us 
quantitative." 

"These instruments allow for inclusion of more information in our 
analysis." 


can only be interpreted in one way," 94.5% of students 
disagreed, with no students agreeing with the statement; 
94.6% of students agreed with the statement, "Even when 
scientific investigations are done correctly, the information 
that scientists discover may change in the future." Students 
responding to the statement, "Scientific research must 
involve experimentation," selected agree 58.2.6% of the 
time, while 36.4% selected disagree, and 5.4% could not 
decide. Finally, 61.8% of students agreed with the statement, 
"If multiple hypotheses can explain the same set of 
observations, the simpler hypothesis is preferred by scien¬ 
tists," while 21.8% disagreed, and 16.4% could not decide. 
Students also select all fields (e.g., biology, chemistry, 
geology) for which they thought X-ray fluorescence analysis 
could be applied to answer research questions (Fig. 4). 

Finally, student responses to the statement, "I can 
imagine the applications of XRF analysis to the following 
fields (select all that apply)," reveal that students had 
changed their ideas about the applicability of XRF analysis 
to different disciplines (Fig. 4E). For biology, mathematics, 
and astronomy, students saw less applicability postlearning 
relative to prelearning, while for all other disciplines shown 
(including geology), a higher percentage of students could 
imagine the application of elemental XRF analysis to 
research questions postlearning relative to prelearning (Fig. 
4E). 

Summative External Evaluation 

Halx (coauthor) was responsible for the collation of 
information summarized in Figs. 3 and 4. His review of 
student free-response information revealed that prior to 
course activities, students had little to no idea about the 
capability of the instruments, while postlearning responses 
suggested a marked increase in engagement directly tied to 
instrument use. Table VII provides a number of samples of 
student free-response data, which support his claim. In fact, 
according to Halx, it was clear that the instruments not only 
enhanced learning, but also made it more fun. 

Halx also addressed our success in achieving the 
learning goals of the funded NSF project, especially with 


regard to the impact of XRF and ICP-OES instrument use on 
those learning goals. Halx posits that these instruments 
engaged students more fully in the process of science, with 
instrument use leading to a more engaged classroom and 
laboratory activity than would have been possible without 
their use. Based on the significant improvement revealed in 
the students' explanations of elemental analysis and 
spectroscopy from the pre- to the postlearning question¬ 
naires, and their answers to embedded exam questions and 
quizzes (as assessed by faculty members), Halx concluded 
that instrument use, when combined with a revised 
curricular context, led to a greater depth of understanding 
of elemental analysis and spectroscopy than previously 
reported student laboratory experiences. 

IMPLICATIONS OF PRE- AND 
POSTLEARNING ASSESSMENT 

We believe that the initiative's three primary objectives, 
listed in "A New Scientific Literacy Initiative" section, were 
successfully met. Next, we discuss the implications of 
student assessment results as they pertain to each of the 
objectives. These inter-related student learning goals can be 
tied directly to Trinity University's understanding natural 
science and technology requirement (see "Study Population 
and Setting" section). 

Objective 1: Improving Student Understanding of the 
Scientific Process and the Nature of Science 

The pre- and postlearning questionnaires reveal the 
most obvious changes in both student attitudes about 
science and perceived understanding of the scientific process 
(Figs. 3 and 4). Interestingly, of the Likert scale statements in 
Fig. 3, student responses to the statement, "I understand the 
scientific method," changed the least, with only a small 
(0.29) increase in the level of agreement with the statement 
from pre- to post-test (Fig. 3A). However, the standard 
deviation dropped 0.20 between pre- and postlearning 
questionnaires, primarily due to a change from 15.3% of 
students selecting disagree or neutral on the prelearning 
questionnaire relative to only 1.9% on the postlearning 
questionnaire. We interpret this as a significant change 
because student perception of the scientific method prior to 
enrolling in the physical geology laboratory course was 
based on the erroneous idea that all scientists engage in a 
static and relatively linear research process. 

In fact, the scientific method construction activity 
revealed that most students thought that all scientists work 
in a similar manner that could only involve experimental 
research, thought that research leads to an inevitable 
"conclusion" (Figs. 5 and 6), and rarely mentioned "theory" 
or "law" in their flow charts. We believe that students were 
affected by a discussion that preceded the activity, which 
likely explains why student flow charts did not include either 
of these terms. Before this activity, instructors led a 
discussion about the stereotyping of scientists. Students, as 
expected, consistently describe a scientist as male, Cauca¬ 
sian, wearing eyeglasses, with disheveled hair, in a white lab 
coat, and holding a beaker. 

Based on student responses on the subsequent activity, 
which in most cases did not include the terms theory or law, 
we believe that students focused on the work of an isolated 
scientist in a laboratory setting. Postactivity discussion 
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revealed that students thought that hypotheses led to 
theories, which when "proven," became laws. Therefore, 
we believe that student responses on the prelearning 
questionnaire did not reveal an understanding of the 
scientific method practiced by scientists in the real world, 
but instead reflected the very linear and static idea 
perpetuated in many textbooks (e.g., McComas, 1998). We 
also believe that our instructions on the handout likely 
impacted student responses (i.e., "explain the scientific 
method using a flow chart to aid your explanation"); we 
chose this phrasing in order to make clear what we were 
asking of students, so that the instructor would not be 
required to explain what we were asking of students 
(thinking, ironically, that an instructor's explanation might 
bias the responses). We consider the relatively small 
numerical increase in student-perceived understanding of 
the scientific method to be much more significant, with the 
postlearning questionnaire indicating that students had a 
better understanding of how science works in the real world 
(Fig. 1), including the basic terminology of science, the 
research methods employed, and the relationships among 
the terms hypothesis, theory, and law. 

This hypothesis is supported by student responses on 
the scientific process quiz, where students addressed 
research methods, the meanings and relationships among 
hypothesis, theory, and law, and other aspects of the 
scientific process (Table VI). By this point in the course 
(week 11; Table III), students had been repeatedly exposed 
to and engaged in activities that related directly to these 
concepts, so this result was not surprising. However, student 
responses on the postlearning questionnaire (week 15; Table 
III) suggest that the high student scores on the scientific 
process quiz were not a true indicator of all students' long¬ 
term retention and learning. 

For some statements related to the scientific method, 
many students' postlearning understanding of some con¬ 
cepts related to the scientific process did not significantly 
change. In Figs. 4A and 4B, student pre- and postlearning 
responses suggest that most students considered good 
scientific data to be open to interpretation both before and 
after learning activities (Fig. 4A), and Fig. 4B suggests that 
students understood that the information revealed by 
scientific research may change in the future, with nearly 
identical results before and after laboratory activities. These 
data reveal that although students initially (prelearning) 
considered the scientific process to be relatively static and 
linear, after taking the course, students understood that 
science is dynamic. Therefore, we do not consider the 
relatively similar pre- and postlearning responses to refute 
an overall improvement of student understanding. 

However, student postlearning responses to the state¬ 
ment, "Scientific research must involve experimentation," 
remained dominated by agree, although there was a 
significant increase in the number of students who 
disagreed with the statement (Fig. 4C). We assume that 
these responses reveal a preconceived notion that is 
difficult to change; the majority of students who partici¬ 
pated in this course still see a scientist as someone who 
must use experimentation to perform research. Finally, 
student postlearning responses to the statement, "If 
multiple hypotheses can explain the same set of observa¬ 
tions, the simpler hypothesis is preferred by scientists," 
revealed that the majority of students understand the 


concept of Occam's razor. However, many students seem 
to consider the choice of hypotheses to be case dependent 
(Fig. 4D). 

Importantly, when given the opportunity to formulate 
their own hypotheses related to real data (ICP-OES analysis 
activity; Table V), students used what they had learned to 
both interpret physical and chemical processes (3 in Table V) 
and develop new, valid hypotheses based on those data (4 in 
Table V). When combined with student responses on the 
second practical exam in the course (Practical Exam II; Table 
III), it also appears that students better understood the 
research methods used by scientists, the relationships 
among hypotheses, theories, and laws, and the role of peer 
review in the process of science (student responses are 
included in the Supplemental Materials for Practical Exam II, 
available at: http://dx.doi.org/10.5408/13-073s3). 

Objective 2: Improving Student Understanding of 
Quantitative and Qualitative Elemental Analyses 

Student responses to Likert scale statements shown in 
Figs. 3B, 3C, and 3D reveal perhaps the most striking and 
quantifiable improvements of student understanding of 
elemental analyses. In response to each statement, the 
average postlearning values increased significantly, with 
significantly lower standard deviations. The distribution of 
response populations displays differences that are statisti¬ 
cally significant, with p values of 0.0032, 6.8 x 10 -7 , and 9.0 
x 10 -12 (p = 1 indicates identical populations) for responses 
to statements in Figs. 3B, 3C, and 3D, respectively. The most 
impressive change in understanding was based on the 
statement, "I understand the basic relationship between 
electron shells and energy," with the lowest average 
prelearning value (of the Likert scale responses) and an 
improvement of 0.99 from the average of neutral (3.10) to an 
average of agree (4.09). These fundamental ideas underlie 
the qualitative and quantitative elemental analyses per¬ 
formed as part of the Trinity rock/soil project. 

The multistage Trinity rock/soil project involved quali¬ 
tative XRF elemental analysis (week 4 activity; Table III) and 
quantitative ICP-OES analysis (week 9 activity; Table III). 
Since very few students had performed any sort of elemental 
analysis prior to enrolling in the physical geology laboratory 
course, we assume that the quality and accuracy of 
postlearning student responses are primarily due to stu¬ 
dents' experiences as part of this course. Based on student 
responses on these activities (Supplemental Materials and 
Table V) and instructor-student interactions, students 
appeared to gain a better understanding about the informa¬ 
tion that elemental analysis can provide; were more 
confident in evaluating trends or patterns in elemental data; 
had improved their abilities to modify and/or formulate 
testable hypotheses; and understood the limitations of 
different types of elemental data. These responses were in 
stark contrast to students' initial attempts to write valid 
research questions (Table V). 

Objective 3: Perceptions of the Applicability of 
Science to Solve Real-World Problems 

The change in student responses to the statement, "I can 
imagine the applications of XRF analysis to the following 
fields (select all that apply)," from the pre- to the 
postlearning assessment revealed a better, more accurate 
understanding of the applicability of XRF elemental analysis 
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to address problems in different fields. For biology, 
mathematics, and astronomy, students saw less applicability 
postlearning relative to prelearning, while for all other 
disciplines shown (including geology), a higher percentage 
of students could imagine the application of elemental XRF 
analysis to research questions postlearning relative to 
prelearning (Fig. 4E). 

In addition, students participated in an authentic soil 
analysis investigation, thinking about the relative impor¬ 
tance of natural and human processes in soil development 
on Trinity University's campus. In the final step of this 
project, students developed valid, testable hypotheses based 
on observational and elemental data (Table V). Based on 
these responses and instructor-student interactions and the 
initial research questions posed (Table IV), we believe that 
postlearning students had a better understanding of how 
science can address real-world problems. These implications 
are supported by the work of Wong and Hodson (2009), who 
advocate the use of authentic scientific practice to provide 
context for student learning. 

A TRANSFERABLE MODEL? 

We believe that this curricular development model, 
grounded in the results of previous educational research, can 
be relatively easily transferred to laboratory courses in 
geosciences programs as well as to laboratory courses in 
other natural sciences. Based on our course design efforts 
and assessment results, we have a series of recommenda¬ 
tions for science educators interested in either improving an 
established course or designing a new one. Although we 
have numbered our recommendations based on our own 
experiences and the work of previous researchers (docu¬ 
mented earlier in this paper), curricular design and revision 
represent a dynamic process that may vary in the sequence 
of steps or in the way that feedback is monitored and/or 
integrated within a given institution or program. Our 
recommendations: 

(1) Outline the primary student learning goals for the 
course. These learning goals should be authored 
with assessment in mind (i.e., can a goal, as written, 
be effectively assessed?) and should be related to 
both institutional (e.g.. Trinity University's general 
education requirements) and program goals. 

(2) Determine how students will demonstrate their achieve¬ 
ment of learning goals. Think carefully about what 
students will produce in the context of the course 
that can be assessed. These assessment tools could 
range from a single question on a quiz or exam to a 
laboratory report to pre- and postlearning ques¬ 
tionnaires. In today's age of assessment, this step 
will likely become part of course design from start to 
finish. It is likely that the integration of assessment 
tools within the curriculum will evolve during the 
course design process, during instruction, and after 
a course has been taught. 

(3) Develop a longitudinal assessment structure that 
permits frequent feedback using a variety of assessment 
tools. By designing a course curriculum with 
frequent assessment using a variety of methods, 
both students and instructors benefit. Students are 


more immediately aware of shortcomings in their 
own understanding, and instructors can identify 
problems with student achievement before these 
problems become more significant. Ideally, pre- 
and postlearning assessment tools, both those 
authored by course designers and those validated 
in the literature (e.g., Aikenhead and Ryan, 1992; 
Bradford et al., 1995; Libarkin, 2001), should be 
used to best evaluate the change in student 
understanding of conceptual or skill-based material. 

(4) Incorporate hands-on participation in the gathering and 
interpretation of data as a significant percentage of 
student activities. As established in previous research 
(e.g., Gagne, 1963; Tobin, 1990; O'Neill and Pol- 
man, 2004) and in this study, student learning 
appears to be impacted when students engage in 
their own scientific data gathering and interpreta¬ 
tion. 

(5) Ensure that iteration is part of the course structure. The 
work of Tobin (1990) and the results from this study 
support a link between student learning and 
retention with the repetition of both skills and 
conceptual materials. 

(6) Explicitly relate all learning opportunities to the 
scientific process. Results from this study reinforce 
the findings of Lederman (2007) and suggest that 
students will develop a deeper understanding of the 
scientific process if course activities are always 
placed within a scientific process framework. The 
work of Carpi and Egger (2010) aided us tremen¬ 
dously in our efforts. 

(7) Integrate opportunities for both critical thinking and 
student reflection. Our study supports the results of 
previous research, suggesting that both critical 
thinking (e.g., Jurecki and Wander, 2012) and 
reflection (e.g., Schwartz et al., 2004) are important 
components of the learning process. With opportu¬ 
nities to critically evaluate and reflect on their own 
activities, we believe that students are more likely to 
recognize their own participation in the scientific 
process and consequently improve their own 
scientific literacy. 

(8) If possible, involve an external evaluator early in the 
course development process. This will make course 
curricular and assessment design a more thoughtful 
and robust process. As course designers, faculty do 
not usually consider how an evaluator outside of 
their department or program would assess the 
achievement of student learning goals or the overall 
success of curricular changes. 

(9) If possible, provide learning opportunities that involve 
the use of research-grade instrumentation. As estab¬ 
lished by previous researchers (Nakhleh and Kraj- 
cik, 1993, 1994; Richter-Egger et al., 2010) and this 
study, there appears to be a positive link between 
the use of this instrumentation and student learn¬ 
ing. We believe that instructors can better engage 
students by providing an explicit scientific process 
context for such a learning opportunity, stressing 
that such an instrument can be used by scientists to 
build on our understanding of physical and/or 
chemical processes in the natural world. 
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CONCLUSIONS 

We aimed to improve the scientific literacy of students in 
the physical geology laboratory course, a 3 h, stand-alone 
course offered by the department of geosciences at Trinity 
University. We used a combination of significant curricular 
revision and the integration of research-grade instrumenta¬ 
tion into class activities to achieve overarching learning goals 
related to the scientific process and the nature of science, 
quantitative and qualitative elemental analysis, and the 
application of scientific research to real-world problems. 
Coupled with these changes, we embedded a wide range of 
assessable learning opportunities related to the scientific 
process throughout the course. We focused primarily on 
prelearning activities early in the course and postlearning 
activities near the end of the course to measure the changes 
in student understanding related to laboratory activities. 

Student data indicate that our efforts have been largely 
successful. These successes support the idea that hands-on 
participation (e.g., Gagne, 1963; AAAS CSE and Gagne, 
1965; O'Neill and Polman, 2004), interpretation, and 
iteration (Tobin, 1990) are key components to effective 
student learning in the sciences. Importantly, we posit that 
by explicitly relating these student learning opportunities to 
the scientific process (as summarized by Carpi and Egger, 
2010), we have provided an effective path toward improved 
student scientific literacy, an idea supported by many 
researchers (e.g., Cashin and Downey, 1995; Smart and 
Ethington, 1995; Lederman, 2007). 

By including activities related to the Trinity rock/soil 
project, we provided students with an opportunity to think 
critically about data and the interpretation of that data, an 
important and commonly overlooked component of the 
scientific process Qurecki and Wander, 2012). In addition, 
student responses reveal that the integration of the XRF and 
ICP-OES into the physical geology laboratory, in the context 
of the Trinity rock/soil project, aided student learning about 
elemental analysis and spectroscopy, an idea proposed by 
researchers in introductory chemistry laboratory courses 
(Nakhleh and Krajcik, 1993,1994; Richter-Egger et al., 2010). 

Although we focused here on the application of these 
ideas to a physical geology laboratory course, we believe that 
this model can be easily transferable to any introductory 
laboratory course that includes scientific literacy as a primary 
learning goal. By integrating course revision, assessment 
framework development, and the application of research- 
grade instrumentation to project-based learning opportuni¬ 
ties, student learning and related scientific literacy should 
inevitably improve. 
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